Metadata is additional data that describes the actual data. For example, metadata can be used in cluster file systems to describe where the corresponding data is stored. Cluster file systems allow multiple client devices to share access to files over a network. For many cluster file systems, especially for High Performance Computing (HPC) applications such as the Linux-based Lustre file system or the General Parallel File System (GPFS) developed by IBM Corp., the metadata servers (MDSs) are a bottleneck that reduce performance gains that would otherwise be available due to the intended parallel data access.
A number of techniques have been proposed or suggested to parallelize metadata access using distributed metadata servers. Unfortunately, when distributed metadata servers access a shared metadata state, the ACID (Atomicity, Consistency, Isolation, Durability) requirements result in complicated locking protocols and multiple round trip communications for coordination to satisfy the two-phase commit protocols required for metadata consistency. See, for example, Giuseppe Congiu et al., “One Phase Commit: A Low Overhead Atomic Commitment Protocol for Scalable Metadata Services,” 2012 IEEE Int'l Conf. on Cluster Computing Workshop (2012), incorporated by reference herein.
A need therefore exists for improved metadata servers that allow parallel metadata access to a shared low latency persistent key-value metadata store.